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ABSTRACT  SYNTHERS  are  cognitive  agents  that  perform  the  job  of  a  team  member,  or  any  other  role  external  to  the 
team,  that  normally  requires  a  human  role-player  or  trainer  to  enable  team  training.  We  are  conducting  a  set  of 
psychological  training  experiments  in  which  teams  are  being  trained  in  a  military  command  and  control  task  with 
confederates  who  play  the  roles  of  SYNTHERS.  By  using  confederates,  we  were  able  to  rapidly  define,  assess  and 
evaluate  the  requirements  for  simulated  teammates,  and  the  relative  contributions  of  certain  components,  such  as 
performance  measurement  and  on-line  feedback,  to  training  outcomes.  Based  on  our  experiments,  we  believe  that 
SYNTHERS,  when  functioning  both  as  teammates  and  as  Intelligent  Tutoring  Systems,  will  not  only  reduce  the  cost  of 
training,  but  also  result  in  increased  training  effectiveness. 

prohibitive  in  today's  climate  of  decreasing  military 
1.  Introduction  budgets  and  manpower. 


Team  training  increasingly  takes  place  in  synthetic 
environments.  However,  such  training  is  often  still 
modeled  after  live  team  training,  resulting  in  a  very  cost¬ 
intensive  process  in  terms  of  both  the  time  and  resources 
required.  Specifically,  gathering  all  of  the  team  members, 
instructors,  and  support  personnel  to  conduct  the  training 
is  very  costly,  and  does  not  even  include  the  costs 
involved  with  the  lost  productivity  associated  with  the 
other  tasks  that  those  personnel  are  not  doing  while 
helping  other  teammates  practice  or  while  practicing 
themselves  [1],  This  issue  becomes  even  more  salient  in 
cases  where  large-scale  training  of  distributed  teams  is 
involved.  For  instance,  the  exercise  "Unified  Endeavor" 
required  470  role-players  with  a  total  of  981 
operators/controllers.  These  numbers  are  simply 


We  believe  that  the  development  of  synthetic  teammates, 
or  ‘SYNTHERS’,  is  a  promising  alternative  to  training 
with  all  human  teammates  and  role -players.  SYNTHERS 
are  cognitive  agents  that  perform  the  job  of  an  unavailable 
team  member,  or  any  role  external  to  the  team  that 
normally  requires  a  human  role-player  or  trainer  to  enable 
training  or  teamwork  practice.  We  believe  that  they  are 
promising,  since  SYNTHERS  may  always  be  available, 
may  be  modeled  after  experienced  training  personnel,  and 
may  therefore  be  more  cost  effective  in  the  long  run. 

The  research  challenge  lies  in  keeping  the  advantages 
associated  with  human  teammates,  while  minimizing  or 
eliminating  the  disadvantages.  Thus,  SYNTHERS  should 
display  the  same  collaborative  and  cooperative  behavior 
typically  associated  with  human  teammates,  while 
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reducing  some  of  the  negative  aspects  of  training  with 
live  teammates,  such  as  the  relative  unpredictability  of 
their  behavior  and  their  fluctuating  availability  for  team 
training.  When  these  research  challenges  are  met,  we 
believe  that  SYNTHERS  will  not  only  reduce  the  cost  of 
training,  but  will  even  result  in  increased  training 
effectiveness. 

However,  there  is  a  gap  that  has  to  be  bridged  between 
this  conception  of  SYNTHERS  and  current 
implementations  of  computer  generated  forces.  A  recent 
National  Research  Council  report  [2]  identified  that  a 
shortcoming  in  the  area  of  military  simulations  is  the  lack 
of  behavioral  realism  in  computer  generated  forces 
(CGF’s).  Most  synthetic  force  models  within  military 
simulations  have  been  constructed  using  relatively 
primitive  human  models  in  which  the  richness  of  behavior 
and  decision-making  are  represented  in  only  a  coarse  and 
brittle  manner.  This  has  produced  simulated  forces  with 
unrealistic  behavior  and  simplistic  responses  that  do  not 
correspond  to  the  behavior  of  real  individual  soldiers  and 
units  [2], [3], 

Therefore,  in  order  to  fulfill  the  objectives  of  our 
SYNTHERS,  there  is  a  need  for  improvement  in  the 
behavioral  representation  of  CGF’s.  Part  of  this 
improvement  will  come  through  the  development  of  new 
architectures  for  cognitive  modeling.  However,  even  if 
very  powerful  architectures  existed  for  the  modeling  of 
SYNTHERS,  the  question  would  still  be  what  behaviors 
they  should  exhibit. 

The  question  of  required  behaviors  is  probably  one  that 
cannot  be  answered  in  general;  Chandrasekaran  and 
Josephson  [4]  argue  that  what  a  cognitive  model  needs  to 
contain  is  vitally  affected  by  what  kinds  of  questions  one 
needs  to  answer,  i.e.  the  goals  of  the  simulation. 
Therefore,  the  fidelity  of  the  model  must  be  judged  by  the 
needs  of  the  model  user.  They  describe  a  research  strategy 
in  which  the  requirements  of  models  are  empirically 
investigated.  Our  approach  towards  the  definition  of 
SYNTHERS  as  teammates  for  team  training  has  been 
somewhat  along  those  lines.  The  goals  of  this  research  are 
threefold: 

•  To  define  empirically  validated  requirements  for 
SYNTHERS  for  team  training.  The  emphasis  is  not 
just  on  defining  requirements  but  also  on  the 
empirical  validation  of  those  requirements:  we  would 
like  to  know  what  the  relative  contribution  of  certain 
SYNTHER  requirements  is  in  terms  of  training 
effectiveness. 

•  To  develop  guidelines  for  the  development  of 
plausible  cognitive  models. 


•  To  develop  guidelines  for  scenario  development 
including  role  players  and/or  SYNTHERS. 

2.  Research  Approach 

In  our  approach  to  this  problem,  a  set  of  psychological 
training  experiments  is  being  carried  out,  in  which  teams 
are  being  trained  with  simulated  teammates  [5].  However, 
these  simulated  teammates  are  not  executable  models  of 
intelligent  agents.  Instead,  we  chose  an  approach  in  which 
humans,  confederates  to  the  experimenter,  play  the  roles 
of  the  simulated  teammates.  By  using  confederates,  we 
are  able  to  rapidly  assess  and  evaluate  what  the 
requirements  of  the  simulated  teammates  should  be,  and 
what  the  relative  contributions  of  certain  components  of 
the  simulated  teammate  add  to  the  training  outcome,  both 
quantitatively  and  qualitatively.  We  are  not  just  interested 
in  defining  requirements  for  synthetic  teammates,  but 
even  more  in  the  empirical  validation  of  those 
requirements.  The  use  of  a  number  of  different  measures 
adds  to  the  strength  of  the  approach.  Using  human 
confederates  instead  of  software  models  at  this  point  in 
the  research  process  has  a  number  of  advantages.  The 
most  obvious  one  is  that  it  is  easier,  and  cheaper,  to 
change  the  behavior  of  human  confederates  through 
scripts  than  to  redesign  a  software  model.  Thus,  in 
working  with  confederates  we  have  chosen  for  a  relatively 
low-cost  approach  for  testing  proposed  requirements  for 
SYNTHERS.  Defining  empirically  validated 
requirements  for  SYNTHERS  is  still  in  its  early  stages. 
Proposed  requirements  still  need  extensive  evaluation  and 
will  probably  need  to  be  altered  based  on  these 
evaluations.  Therefore  the  costs  of  developing  software 
models  at  this  point  outweigh  the  benefits.  Second, 
training  experiments  with  human  confederates  aid  the 
exploration  of  other  factors  of  interest  in  the  development 
of  SYNTHERS  for  team  training,  such  as  personality 
variables  and  factors  such  as  facial  expressions,  speech 
intonations  etc.  [6]. 

The  testbed  that  we  use  is  a  modified  version  of  the 
Dynamic  Distributed  Decision-Making  (DDD) 
simulation.  This  computerized  task  (see  Figure  1) 
simulates  a  military  command  and  control  task  in  which  a 
team  of  decision  makers,  each  of  whom  controls 
specialized  sub-platforms,  needs  to  enforce  a 
“demilitarized  zone”.  This  demilitarized  zone  is 
threatened  by  an  enemy,  but  is  also  traversed  by  friendly 
ground  vehicles  and  aircraft.  The  goal  of  the  team  is  to 
identify  the  nature  of  all  vehicles  and  aircraft  (i.e., 
friendly  versus  enemy)  that  encroach  or  enter  the 
demilitarized  zone,  and  then  to  attack  enemy  targets  that 
enter  this  zone,  while  not  harming  any  friendly  vehicles 
and  aircraft.  The  DDD  simulation  was  initially  developed 
by  Serfaty  and  Kleinman  [7],  and  has  been  used  in  a 


variety  of  contexts  and  for  a  variety  of  research  questions, 
including  optimizing  team  organization,  optimizing  team 
decision  making,  and  team  training. 

The  DDD  is  flexible,  in  that  the  number  of  team 
members,  the  lay-out  of  the  area,  the  allocation  of  sensors 
and  weapons,  and  control  over  the  timing,  heading,  path 
and  number  of  targets  approaching  the  demilitarized  zone, 
are  under  the  experimenter's  control.  In  addition,  DDD 
offers  numerous  opportunities  for  the  experimenter  to 
automatically  measure  team  performance  on  a  number  of 
dimensions. 


Figure  1.  The  DDD  testbed 

In  our  initial  experiments,  we  are  using  a  three-person 
team  adaptation  of  the  DDD.  This  team  consists  of  two 
confederates,  and  one  subject  to  be  trained  on  teamwork 
skills.  The  subject  is  not  aware  of  the  fact  that  the  other 
two  team  members  are  in  fact  confederates.  The 
confederates  act  according  to  a  tight  script,  and  their  task 
varies  over  the  experiments  from  just  providing 
opportunities  for  practicing  teamwork  skills,  to  modeling 
those  skills,  and  providing  coaching  and  feedback,  or 
combinations  of  those  behaviors.  This  situation  will  be 
contrasted  with  a  situation  in  which  three  naive  subjects 
are  brought  into  the  experiment,  reflecting  the  “classical” 
situation  in  which  someone  is  being  trained  with  his/her 
fellow  trainees.  Although  the  DDD  can  be  made 
sufficiently  realistic  for  military  personnel  to  exhibit  their 
skills,  we  are  using  university  undergraduates  as  subjects. 
This  enables  us  to  carry  out  the  experimentation  in  a 
relatively  limited  amount  of  time. 

3.  Results 

We  started  our  research  by  defining  a  framework  for 
training  teams  with  synthetic  teammates.  In  recent  years, 
it  has  been  shown  that  a  good  approach  to  training  teams 
with  complex  training  technology  is  linking  training  goals 
to  events  in  training  scenarios  in  a  controlled  fashion. 


This  is  called  the  ‘event-based  approach  to  training’ 
(EBAT)[8j.  The  primary  goal  of  EBAT  is  to 
systematically  provide  opportunities  for  a  training 
audience  to  develop  critical  competencies  by  receiving 
practice  in  simulated  environments  that  are  representative 
of  actual  operational  conditions,  and  receiving  feedback 
linked  to  specific  events  that  occur  during  training. 
EBAT  tightly  links  trainee  needs,  critical  tasks,  learning 
objectives,  scenario  design,  performance  measurement, 
and  feedback.  The  general  assumption  of  EBAT  is  that 
without  a  systematic  linkage  among  these  components 
there  is  no  way  of  knowing  or  ensuring— with  any  degree 
of  certainty— that  the  exercise  training  will  have  its 
intended  effect. 


Figure  2  shows  how  the  EBAT  framework  supports  the 
design,  development,  execution,  and  evaluation  of 
exercises. 


Figure  2.  Components  of  an  EBAT  Framework.  Adapted 
from  [9] 


The  EBAT-approach  provides  a  good  starting  point  for 
the  development  of  a  conceptual  framework  for  training 
teams  with  synthetic  teammates.  It  shows  us  where 
synthetic  teammates  could  possibly  play  a  role  in  training 
teams:  they  could  act  as  fellow  teammates,  as  collateral 
forces,  or  as  enemy  forces.  Second,  it  shows  that,  as 
much  as  scenario  events  are  scripted,  synthetic  teammates 
should  behave  in  a  scripted  manner,  too.  This  does  not 
mean  that  their  behavior  has  to  be  strictly  ‘canned’,  and 
predictable  for  the  trainee,  but  it  does  imply  that  their 
behavior  is  predictable  for  the  scenario  designer. 

The  results  of  our  initial  analyses  show  that  at  least  three 
design  components  can  be  distinguished  for  simulated 
teammates: 

1.  A  task  component:  the  level  of  proficiency  in  the  task 
the  simulated  teammate  should  have.  Our  analysis 
shows  that  simulated  teammates  do  not  necessarily 
have  to  be  experts  in  the  task,  but  should  master  the 
task  to  such  an  extent  as  to  be  believable  and  useful 
teammates  in  a  team  training  context.  This  will 


probably  involve  some  level  of  both  taskwork  (the 
knowledge  and  skills  involved  in  carrying  out  the 
individual  task)  and  teamwork  (the  knowledge  and 
skills  involved  in  cooperating  with  others)  [10].  An 
analogy  from  the  sports  arena  will  help  to  clarify  this 
issue.  Tennis  instructors,  playing  and  practicing  with 
you  from  the  other  side  of  the  net,  do  not  necessarily 
have  to  be  expert  players  to  provide  you  with  the 
right  shots  to  practice  your  skills.  Along  similar  lines, 
simulated  teammates  do  not  necessarily  have  to  be  at 
an  expert-level  of  proficiency  in  the  task  to  provide 
valuable  instruction  and  practice  opportunities.  In 
fact,  our  data  show  that  our  confederates  are  not 
playing  at  an  expert-level  of  proficiency:  when  they 
are  allowed  to  play  the  DDD  in  a  free-play  situation, 
they  perform  much  better  than  when  they  are  playing 
the  confederate-role. 

It  is  impossible  to  script  the  behavior  of  the  simulated 
teammates  for  every  target  in  the  scenario.  It  would 
result  in  very  unrealistic  behavior  with  the  same 
“brittleness”  that  is  critized  in  current  CGF’s. 
Therefore,  we  decided  to  define  a  set  of  “standard 
behaviors”.  For  the  DDD  testbed,  we  defined  the  task 
component  as: 

•  The  synthetic  teammates  staying  in  a  predefined 
area  of  the  screen,  unless  called  upon  by  the 
trainee  or  by  the  script. 

•  The  synthetic  teammate  playing  the  game  in  a 
normal  fashion:  detecting,  identifying,  and 
engaging  targets  that  would  appear  in  their  area. 

•  The  synthetic  teammates  not  communicating 
unless  they  had  to  (when  there  was  a  target  in 
their  area  for  which  they  lacked  the  attacking 
power),  or  when  they  were  asked  a  specific 
question. 

•  The  synthetic  teammates  obeying  the  requests 
from  the  trainee. 

•  The  synthetic  teammates  following  a  time-based 
script,  in  which  they  would  take  certain  actions 
(verbal  or  game -based)  upon  certain  targets  or 
their  sensor  and  weapon  systems.  The  events  in 
the  scripts  are  related  to  the  instructional/practice 
component  described  in  the  next  paragraph. 

An  instructional/practice  component:  what  are 
appropriate  instructional  and  practice  strategies  for 
this  teammate  to  exhibit.  One  of  the  most  important 
features  and  potential  benefits  of  training  with 
simulated  teammates  is  that  they  can  demonstrate 
certain  behaviors  in  training  and  provide  systematic 
and  structured  opportunities  for  practice.  For 
example,  providing  backup  to  other  team  members  in 
case  they  become  overloaded  is  regarded  as  an 


important  teamwork  skill  [11].  In  situations  where 
you  are  being  trained  in  an  expert  team,  however, 
members  do  not  become  easily  overloaded,  and  the 
opportunities  for  practicing  this  teamwork  skill  are 
rare.  Flowever,  in  training  with  a  simulated 
teammate,  these  opportunities  do  exist,  because 
sythetic  teammates  can  be  specifically  scripted  to 
become  overloaded  through  a  combination  of 
external  events,  and  the  reaction  of  the  simulated 
teammate  to  this  situation.  Therefore,  training  with 
simulated  teammates  provides  the  trainee  with 
excellent  opportunities  for  practice  that  might  hardly 
exist  otherwise. 

The  events  in  the  scripts  that  would  result  in 
opportunities  for  practice  were  based  on  a  cognitive 
task  analysis  of  supporting  behavior  in  the  DDD.  In 
one  of  our  initial  experiments,  we  focused  on  the 
training  of  supporting  behavior,  which  consists  of 
three  components: 

•  Providing  backup  to  other  teammates  when  they 
become  overloaded 

•  Requesting  backup  in  situations  in  which  you 
become  overloaded 

•  Catching  and  correcting  errors  before  they  have 
negative  consequences  on  the  teams’ 
performance. 

A  cognitive  task  analysis  focused  upon  these  three 
components,  and  when  and  how  they  would  show  up 
in  the  DDD.  By  carefully  observing  both  expert  and 
relatively  novice  players,  we  observed,  for  example, 
that  the  following  errors  are  quite  ‘natural  errors’  in 
the  DDD: 

•  Attacking  an  enemy  in  the  neutral  zone 

•  Relying  on  incorrect  identification  data  and 
therefore  attacking  targets  mistakenly 

•  Prioritization  errors:  giving  priority  to  targets 
which  are  still  less  threatening  than  other  targets, 
since  they  are  further  removed  from  a  predefined 
no-fly  zone. 

•  Attacking  with  a  weapon  system  which  is  out  of 
ammunition. 

The  scripting  of  these  ‘natural  errors’,  in  combination 
with  the  “standardized  behavior”  as  described  in  the 
previous  paragraph,  resulted  in  believable  SYNTF1ER 
behavior.  The  data  show  no  indication  of  participants  who 
questioned  the  realism  of  the  SYNTFIERS.  This  can  be 
taken  as  evidence  that  SYNTFIERS  in  this  context  do  not 
have  to  perform  at  an  expert  level,  and  can  be  scripted 
without  being  predictable  or  believable  for  the  trainee. 

3.  A  performance  measurement/feedback  component: 
Our  research  has  led  us  to  question  whether  it  will  be 
possible  to  measure  teamwork  skills  on-line  and 


automatically  in  dynamic  environments.  The  results 
show  that  this  is  possible  to  some  extent,  at  least  in 
the  environment  that  we  are  currently  working  in,  the 
DDD  simulation.  The  generalizability  of  these  results 
to  real-life  military  environments  is  something  which 
still  needs  to  be  addressed. 

4.  Implications 

One  of  the  implications  of  this  line  of  reasoning  is  that 
we  propose  a  layered  model  of  scenario  events  for 
defining  the  behavior  of  synthetic  teammates  in  team 
training.  The  first  layer  consists  of  the  ‘normal’  scenario 
events  that  occur:  the  behavior  of  entities  external  to  the 
team  (such  as  a  target  approaching  with  a  certain  speed  at 
a  certain  altitude). 

A  second  layer  consists  of  the  behavior  of  the  synthetic 
teammate(s):  How  should  synthetic  teammates  behave 
during  these  events?  For  example,  if  the  training  goal  is 
to  learn  about  providing  backup  to  another  team  member, 
the  training  exercise  must  create  an  opportunity  to 
provide  backup.  In  the  DDD  this  can  be  done  by  sending 
a  wave  of  targets  to  one  simulated  team  member  —  too 
many  to  act  upon  by  him/herself  — ,  and  have  this 
teammate  ask  for  help.  However,  this  opportunity  for 
providing  backup  will  probably  only  be  noticed  by  the 
trainee  if  he/she  is  not  also  overloaded.  Therefore,  we 
have  to  manipulate  the  scenario  to  ensure  that  our  trainee 
is  not  overloaded.  Finally,  we  can  make  the  cue  for 
providing  backup  even  stronger  if  it  is  clear  to  the  trainee 
that  the  third  teammate  on  the  team  cannot  help  out;  for 
example,  since  most  of  his/her  weapon  or  sensor  systems 
have  run  out  of  ammunition  or  fuel. 

A  third  layer  consists  of  training  strategies  linked  to  team 
performance  measurement  and  providing  on-line 
feedback. 

All  three  layers  are  connected  by  their  relationship  to 
training  objectives/competencies,  but  they  will  each 
contribute  to  achieving  these  goals  in  different  ways.  The 
first  layer  defines  events  external  to  the  team.  The  second 
layer  defines  when  and  how  the  synthetic  teammates  will 
provide  opportunities  for  training  certain  (teamwork) 
skills.  The  third  layer  defines  how  intelligent 

performance  evaluation  and  on-line  feedback  could  be 
provided  by  a  synthetic  teammate.  The  three  layers  can 
be  linked  in  various  ways:  by  time  (when  will  a  certain 
event  at  each  of  the  three  layers  occur),  by  the 
contingency  of  events  (e.g.,  the  synthetic  teammate  will 
provide  an  opportunity  for  practicing  error  correction  at 
specific  targets),  or  by  a  combination  of  both. 

Another  important  implication  of  this  conceptual 
framework  and  supported  by  empirical  results,  is  that 


synthetic  teammates  do  not  necessarily  have  to  behave  as 
experts  in  the  task.  By  redefining  SYTHERS  as  entities 
that  serve  certain  training  goals,  their  role  is  not 
necessarily  one  of  being  a  ‘normal’,  proficient  teammate. 
Therefore,  the  mission  of  synthetic  teammates  could  be  to 
provide  opportunities  for  practicing  certain  skills,  instead 
of  being  expert  performers  in  the  task  itself.  It  is 
important  to  put  the  trainee  in  the  center  of  attention,  and 
not  so  much  the  SYNTHER,  since  their  goal  is  just  to 
help  train  the  trainee. 

5.  The  road  ahead:  SYNTHERS:  fusion 
between  SAF  and  ITS 

Our  research  focuses  upon  empirically  validated 
requirements  for  SYNTHERS  in  team  training.  The  initial 
driving  force  behind  this  research  was  the  idea  to  have 
SYNTHERS  fill  in  the  roles  of  missing  teammates  in 
team  training.  However,  in  the  course  of  the  research,  it 
became  clear  that  SYNTHERS  have  more  to  offer.  They 
can  be  assigned  various  roles:  as  teammates,  as  collateral 
forces,  or  as  opposing  forces.  More  interestingly,  they  can 
be  assigned  a  role  in  the  training  process  itself.  They  can 
act  as  coaches,  who  provide  opportunities  for  practicing 
certain  skills,  and  can  have  a  role  in  diagnosing, 
evaluating,  and  remediating  trainees’  knowledge  and  skill. 
Through  the  integration  and  fusion  of  SYNTHERS  as 
SAF’s  and  SYNTHERS  as  ITS’s,  we  will  be  able  to 
explore  a  new  dimension  of  learning  companions[12],  and 
pedagogical  agents  [13]. 
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